Photo-image-based 3D modeling system on a mobile device

ABSTRACT

A system that runs in web browsers of mobile devices that allows mobile users to take photos of building exteriors and interiors or other real world objects, upload photos, share photos with others, and use the photo images to model the 3D models with the system&#39;s image-based modeling interface.

BACKGROUND

1. Field of the Invention

This field is generally related to the use of photographic informationin conjunction with a simulated three-dimensional (3D) environment.

2. Related Art

Three-dimensional modeling tools and other computer-aided design (CAD)tools enable users to define three-dimensional models, such as athree-dimensional model of a building. Photographic images of thebuilding may be available from, for example, satellite, aerial,vehicle-mounted street-view and user cameras. By texture mappingphotographs of the buildings onto the three-dimensional models withphotogrammetry algorithms, it becomes possible to use multiplephotographs to create a visually appealing three-dimensional model.

Mobile devices with integrated cameras represent a valuable source ofimage data. The potential value of the image data is increasedconsiderably because the image data may be associated with camera modelinformation including location and orientation data from a locationsensor, such as a GPS sensor.

BRIEF SUMMARY

Embodiments relate to using photographs, camera model information, anduser input from mobile devices to create and navigate image-basedthree-dimensional models.

In a first embodiment, a computer-implemented method is provided foraltering a three-dimensional model based on a set of two-dimensionalimages at a mobile device including a first camera. Thecomputer-implemented method includes several stages, from (a) to (g).Stage (a) includes capturing, with the first camera, a localphotographic image with a first field of view. Stage (b) includesdetecting, using a location sensor coupled to the mobile device, a firstlocation and a first orientation of the first camera when the localphotograph was captured in (a), wherein a first camera model includesthe first location and the first orientation. Stage (c) includes sendinga request to a server, separate from the mobile device, for anadditional photographic image captured from a second camera remote fromthe mobile device, the additional photographic image having a secondfield of view overlapping with the first field of view. Stage (d)includes receiving, from the server, the additional photographic imageand a second camera model including a location and an orientation of thesecond camera when the additional photographic image was captured. Stage(e) includes receiving a user constraint input by a user, the userconstraint associating a position on the local photographic image with aposition on the three-dimensional model. Stage (f) includes applying aphotogrammetry algorithm to alter the three-dimensional model based, atleast in part on, the first and second camera models and the one or moreuser constraints. Stage (g) includes determining whether the user wantsto add more constraints, and, if so, repeating stages (e) and (f).

Other embodiments describing systems and computer readable medium foraltering a three-dimensional model based on a set of two-dimensionalimages at a mobile device including a first camera are also disclosed.

Further embodiments, features, and advantages of the invention, as wellas the structure and operation of the various embodiments of theinvention are described in detail below with reference to accompanyingdrawings.

BRIEF DESCRIPTION OF THE FIGURES

The accompanying drawings, which are incorporated herein and form a partof the specification, illustrate the present invention and, togetherwith the description, further serve to explain the principles of theinvention and to enable a person skilled in the pertinent art to makeand use the invention.

FIG. 1 is a diagram showing an exemplary mobile device for constructingthree-dimensional models, as in an embodiment.

FIG. 2 is a flowchart showing a method for using a mobile device toprovide photography, camera models, and user input that allow the userto create a three-dimensional model.

FIG. 3 is a diagram showing how a user and a plane may obtain imageryfor use in the context of a three-dimensional model.

FIG. 4 is an example local photo, taken facing the front of a building.

FIG. 5 is an example remote photo, taken flying over a building.

FIG. 6 is a screenshot of a mobile device interface that allows a userto organize views of a building to be transformed into athree-dimensional model.

FIG. 7 is a screenshot illustrating a three-dimensional model thatresults after mapping various two-dimensional views onto the sides of athree-dimensional solid model.

FIG. 8 is a screenshot that shows how various two dimensional images maybe dragged onto the side of a three-dimensional solid to texture mapthem onto the sides of the solid.

FIG. 9 is a screenshot that shows how the corners of a two-dimensionalimage may be mapped onto the corners of a three-dimensional solid toprovide for texture mapping the two-dimensional image onto the solid.

FIG. 10A illustrates how stereo cameras may be used to obtain 3Dphotography.

FIG. 10B illustrates how a 3D model may be depicted on a 3D screen.

FIG. 11 illustrates an interface for a building modeler, in anembodiment.

The drawing in which an element first appears is typically indicated bythe leftmost digit or digits in the corresponding reference number. Inthe drawings, like reference numbers may indicate identical orfunctionally similar elements.

DETAILED DESCRIPTION OF EMBODIMENTS

As mentioned above, many mobile devices provide camera and locationdetection capabilities. Embodiments take maximum advantage of theinformation provided by a mobile device for image-basedthree-dimensional modeling. Particularly, embodiments use photographicdata, camera model information, and user input to help the user in bothconstructing the model and in navigating the model.

FIG. 1 is a diagram showing an exemplary mobile device for constructingthree-dimensional models, as in an embodiment. In embodiments, a systemprovides for using a mobile device 100 to alter a three-dimensionalmodel of an environment.

Mobile device 100, in general, will be a portable electronic device withone or more processors 102, a memory 104, a persistent storage 106, oneor more cameras 130A, 130B . . . 130N, and a touch screen 180 andbuttons 184 which allows a user 182 to access and control mobile device100.

Many types of electronic devices may serve as mobile device 100. Suchdevices may include handheld devices, such as cell phones, smartphones,PDAs, gaming consoles, music players, video players, and so on. Other,slightly larger mobile devices such as tablets, GPS devices, and so onmay also be used in certain embodiments.

Mobile device 100 runs an operating system 120 that coordinates therelationship between the software and hardware components of the mobiledevice. A system bus (not shown) may interconnect the various hardwarecomponents of mobile device 100. Alternatively, any hardwareinfrastructure that allows the various components of mobile device 100to communicate with each other may be used in embodiments in lieu of asystem bus.

While these various features of mobile device 100 have been set forth soas to characterize the mobile device and provide for different hardwareaspects of various exemplary system embodiments, it is to be noted thatembodiments need not incorporate all of these features and some mayclearly be implemented in a variety of ways. For example, persistentstorage may be implemented as a hard drive in some embodiments, flashmemory in other embodiments, or may not be present in other embodiments.

Embodiments process information with respect to the three-dimensionalmodel by using a front-end and a back-end. The front-end may run, forexample, in a web browser 134 of the mobile device, or in a dedicatedimage gathering application 132 on the mobile device. The user can takephotos, such as of exteriors or interiors of buildings or other objects,and then use an mobile 3D model application 136 at mobile device 100 tocreate the three-dimensional building or three-dimensional objects. Thefront-end includes the functionality that operates at mobile device 100,where user 182 interacts directly with the local environment of mobiledevice 100 and alters a 3D model using mobile 3D model application 136.

In mobile 3D model application 136, the user only needs to associatetwo-dimensional constraints on the photo images to match the photos withthe geometry. The association, is discussed in greater detail below inassociation with FIG. 9. The textured three-dimensional models areautomatically constructed with a photogrammetry solver that texture mapsthe photo image onto the surfaces of the 3D model. The photogrammetrymay be performed at the front-end in mobile photogrammetry module 144(see FIG. 1B), or at the back-end in server photogrammetry module 160.Whichever photogrammetry module is used, it is used in conjunction withmobile 3D model application 136 to provide data that helps mobile 3Dmodel application 136 display surfaces in the model with photographsappropriately texture-mapped onto them. While mobile photogrammetrymobile 144 is shown as separate from mobile 3D model application 136, inother embodiments it may be included as part of mobile 3D modelapplication 136.

Mobile device 100 is also generally connected to a back-end, which may,for example, be a server 180 that is connected over a wirelessconnection 194 to mobile device 100. The back-end uses the enhancedprocessing and storage capabilities of server 180 to host enhancedcapabilities that user 182 can access remotely using mobile device 100.Server 180 includes a number of modules that organize and manipulate thedata provided by mobile device 100 and uses them in combination withother sources of data to provide support for the three-dimensionalmodel. For example, server 180 may include local image storage 140 tostore and organize photographs that the user takes locally, as well asadditional image storage 142 to store and organize photographs fromother sources. Additional image storage 142 may include shared photosfrom other users, as well as information from sources such assatellites, aerial photography, and photography from land vehicles.

Local image storage 140 and additional image storage 142 are connectedwithin server 180 to a server photogrammetry module 160. Serverphotogrammetry module 160 takes the two-dimensional image informationand uses an appropriate algorithm to texture map the two-dimensionalimage information onto the three-dimensional solid. At server 180, aserver three-dimensional mode navigation module 170 interacts with theinformation provided by server photogrammetry module 160 to provide theuser with access to a three-dimensional interactive experience thatreflects a simulation of the environment that the user has characterizedby mapping images onto surfaces.

It is to be noted that while in one embodiment, the photogrammetry isperformed at server 180 by combining local imagery from mobile device100 with images from additional image storage 142, another embodimentmay provide that the image gathering application 132 or the web browser134 at mobile device 100 may retrieve images from additional imagestorage 142 wirelessly or simply use a set of images that has beenobtained locally, that the image gathering application 132 or webbrowser 134 may provide photogrammetry locally at mobile photogrammetrymodule 144 (see FIG. 1B) the functionality of server photogrammetrymodule 160 and server three-dimensional mode navigation module 170locally to the mobile device at mobile 3D model application 136.

As mentioned above, FIG. 1 further illustrates a mobile photogrammetrymodule 144, user constraint module 138, a server image request module152, and a server image receipt module 154. Mobile photogrammetry module144 carries out photogrammetry tasks at mobile device 100.

User constraint input module 138 enables user 182, using one or both ofbuttons 174 and touch screen 180, to input constraints to determine thethree-dimensional model from two-dimensional images usingphotogrammetry. Each constraint indicates that a position on atwo-dimensional photographic image corresponds to a position on athree-dimensional model.

In embodiments, altering the three-dimensional model may include usinguser constraint input module 138 to add additional user constraints bymapping new user constraints to positions on the three-dimensionalmodel, or moving the user constraints to new positions on thethree-dimensional model. Altering the three-dimensional model caninclude changing the geometry of the three-dimensional model based onthe constraints.

Server image request module 152 initiates downloads of remote images foruse by mobile photogrammetry module 144, and server image receipt module154 receives the content of the remote images.

Furthermore, the user can upload the photos to the back-end and sharephotos with other users to let them model the 3D buildings or objects.When modeling, users can add three-dimensional primitives. Userconstraint input module 138 allows association of images withconstraints with respect to the three-dimensional primitives tofacilitate the photogrammetry. This capability results in a community inwhich modelers can help each other to gain photo images for modeling thereal world in 3D.

Each of the constituent parts of the front-end and back-end may beimplemented in hardware, software, firmware, or any combination thereof.Furthermore, each of the information storage parts of the front-end andback-end may be stored in any type of structured memory, including apersistent memory. In examples, such a persistent memory may beimplemented as a database, including a relational database.

In the detailed description of embodiments that follows, references to“one embodiment”, “an embodiment”, “an example embodiment”, etc.,indicate that the embodiment described may include a particular feature,structure, or characteristic, but every embodiment may not necessarilyinclude the particular feature, structure, or characteristic. Moreover,such phrases are not necessarily referring to the same embodiment.Further, when a particular feature, structure, or characteristic isdescribed in connection with an embodiment, it is submitted that it iswithin the knowledge of one skilled in the art to effect such feature,structure, or characteristic in connection with other embodimentswhether or not explicitly described.

Computer-readable medium embodiments may include any physical mediumwhich is capable of encoding instructions that may subsequently by usedby a processor to implement methods described herein. Example physicalmedia may include floppy discs, optical discs (e.g. CDs, mini-CDs, DVDs,HD-DVD, Blu-ray), hard drives, punch cards, tape drives, flash memory,and memory chips. However, any other type of tangible, persistentstorage that can serve in the role of providing instructions to aprocessor may be used to store the instructions in these embodiments.

Overview of the Method

FIG. 2 is a flowchart of a method 200 using a mobile device to providephotography, camera models, and user input that allow the user to createand interact with a three-dimensional model. Method 200 enablesconstruction of a three-dimensional model based on a set oftwo-dimensional images at a mobile device including a first camera.Method 200 includes several stages, from 210 to 270. The methodpresented here reflects the embodiment in which the user may add anarbitrary number of constraints, and other embodiments may includeconstraints associated with one local image and one additional image.While for clarity various stages are described with respect tocomponents of system 100, a person of ordinary skill in the art wouldrecognize that method 200 may be used in other contexts.

Stage 210 includes capturing, with the first camera, a localphotographic image with a first field of view. Stage 210 may be carriedout using first camera 130A and image gathering application 132.

Stage 220 includes detecting, using a location sensor coupled to themobile device, a first location and a first orientation of the firstcamera when the local photograph was captured in 210. The first locationand the first orientation may be included in a first camera model. In anillustrative example, stage 220 may be carried out using first camera130A and location sensor 150.

Stage 230 includes sending a request to a server, separate from themobile device, for an additional photographic image captured from asecond camera. The second camera may be remote from the mobile device,and the additional photographic image may have a second field of viewoverlapping with the first field of view. Stage 230 may be carried outusing wireless connection 194, server 180, server image request module152, and additional image storage 142. In an example, stage 230 mayinvolve retrieving other images geocoded with a predefined radius of thelocation detected from location sensor 150.

Stage 240 includes receiving from the server the additional photographicimage and a second camera model. The second camera model may include alocation and an orientation of the second camera when the additionalphotographic image was captured. Stage 240 may be carried out usingwireless connection 194, server 180, server image receipt module 154,and additional image storage 142.

Stage 250 includes receiving a user constraint input from a user. Theuser constraint may be associated with a position on the localphotographic image with a position on the three-dimensional model. Stage250 may be carried out using image gathering application 132 and userconstraint input module 138.

Stage 260 includes applying a photogrammetry algorithm to alter thethree-dimensional model based, at least in part on, the first and secondcamera models and the one or more user constraints. Stage 260 may becarried out at server photogrammetry module 160 and serverthree-dimensional mode navigation module 170, or alternatively locallyto mobile device 100 at mobile photogrammetry module 144 and mobile 3Dmodel application 136. The photogrammetry process is discussed ingreater detail below.

Stage 270 includes determining whether the user wants to put moreconstraints, and if so, repeating stages 250 and 260. Stage 270 may becarried out using user constraint input module 138.

The computer-implemented method offers several advantages over priorapproaches to 3D modeling. One advantage is that it facilitates easyintegration of images acquired using cameras local to a mobile deviceinto a 3D modeling environment. Embodiments are designed to allow usersto take pictures of their surroundings and integrate them into 3D modelswith a minimum of effort. Another advantage is that it obtains andassociates camera model data including location and orientationinformation with the local photographs, thereby providing initialinformation to help incorporate the photographs into the model. Current3D modeling technologies do not use location information, but thelocation information is used by embodiments to help facilitate theprocess of setting constraints. Because embodiments have camera modelinformation, this information allows embodiments to provide users withan initial estimate of where constraints should go, saving the user timeand effort. The method also facilitates combining local and additionalphotography, so that multiple sources can easily be used to provide themost versatile and wide-ranging set of photographs possible.

Constructing the Model

FIG. 3 is a diagram showing how a user and a plane may obtain imageryfor use in the context of a three-dimensional model. While in FIG. 3, auser 182 and an airplane 302 are provided as potential photographysources, it should be noted that additional users (not shown) as well asother image sources such as land vehicles and satellites mayadditionally contribute photographic images.

In FIG. 3, user 182 is positioned near a building 310. Building 310, asmay be seen from FIG. 3, is modeled as a rectangular solid, with severaldistinct faces and a chimney 340. User 182 uses camera 130A coupled to amobile device to take a picture of building 310 from a frontalperspective. This frontal perspective shows a door 320, a left window330 and a right window 332, and a chimney 340. This frontal perspectiveis recorded as a local photograph 400 (see FIG. 4), discussed below.

Additionally, an airplane 302 flies over building 310. It uses anairplane camera 304 to take an overhead photograph of building 310 asremote photo 500 (see FIG. 5). Airplane 302 provides an overheadperspective of chimney 340. Airplane 302 may also include an airplanelocation sensor 306 to associate location information with photographstaken by airplane camera 304.

Several pieces of information are recorded when a picture is taken. Whena picture is taken, the camera records an image that is a visualreproduction of its field of view. However, when user 182 takes apicture, location sensor 150 records information for a camera model thatsubsequently provides server photogrammetry module 160 with contextrelated to the picture. The camera model includes a location and anorientation of the picture. The location information generally consistsof a latitude, a longitude, and altitude and provides for a geographicallocation with which the picture is associated. Alternatively, thelocation information may be specified in relationship to geographicalfeatures (i.e. street names or landmarks.) The orientation reflects theposition of the camera with respect to gravity, and helps provideinformation about where the top and bottom of the photograph arelocated. This camera model information helps provide serverphotogrammetry module 160 with an initial guess about how images shouldbe positioned to integrate them into the 3D model. In general, thecamera model suggests an initial placement and orientation ofphotographs that server photogrammetry module 160 can use. By takingadvantage of overlap between multiple photographs, camera models andphotogrammetry modules can use information from multiple photographs andmultiple camera models to fill in gaps from other photographs and cameramodels.

FIG. 4 is an example local photo, taken facing the front of a building.FIG. 4 shows a façade of building 310, including a door 320, twowindows, left window 330 and right window 332, and a chimney 340. Thefaçade is displayed as a local photo 400. Local photo 400 may be taken,for example, as in FIG. 3 by user 182 using camera 130A when they arepositioned in front of building 310. Additionally, local photo may beassociated with a camera model provided by location sensor 150 that iscoupled to mobile device 100. As discussed above, this camera modelwould provide for the location and orientation of the camera which wasused to take the photograph. In this case, it would indicate thelocation of user 182, and that the camera was pointed directly atbuilding 310, with the top of the building at the top of the picture.This camera model information can then be processed appropriately byserver photogrammetry module 160 or mobile photogrammetry module 144 tointegrate the photo into the 3D model at 3D model application 136. Bydefault, the camera model would provide for mapping the photo onto thefront façade of building 310, based on the location and orientationassociated with camera 130A when the picture was taken.

FIG. 5 is an example remote photo, taken flying over a building. FIG. 5shows an overhead remote photo 500 view of the roof for building 310,including an overhead view of chimney 340. As noted previously, theremote photo may provide for an alternative view of the subject of localphoto 400. In the case of remote photo 500, airplane 302 obtains thephoto, and airplane location sensor 306 provides camera modelinformation that can suggest to a server photogrammetry module 160 ormobile photogrammetry module 144 how to integrate the photo into the 3Dmodel at 3D model application 136. In this case, it would be appropriateto map the remote photo onto the top of building 310, because airplanelocation sensor would indicate that the photo was taken from above thebuilding and it was oriented looking down.

FIG. 6 is a screenshot of a mobile device interface that allows a userto organize views of a building to be transformed into athree-dimensional model. For example, a variety of local photos 400 anda remote photo 500 that show ground-based views and an overhead view ofbuilding 310 are displayed on touch screen 180. These photos can beselected for manipulation as in FIGS. 8 and 9 to set constraints thatgovern how the photos are aligned with respect to the 3D model. Mobiledevice 100 additionally includes buttons 184 that a user 182 may use tointeract with the views, as well as a create 3D model button control 610that user 182 may use on touch screen 180 to create thethree-dimensional model. While the button control 610 is depicted as atouch screen 180 button, it may be a physical button as well. Similarly,physical buttons 184 may have functionality that allow user 182 tonavigate and control the mobile device 100 interface, but these physicalbuttons 184 may additionally be provided for as button controls on thetouch screen 180. The buttons 184 may include functionality such asactivating a menu, navigating to a prior screen, navigating to a homescreen, or performing a search, in various embodiments.

FIG. 7 is a screenshot of an interactive three-dimensional model thatresults after mapping various two-dimensional views onto the sides of athree-dimensional solid model 710. Three-dimensional solid model 710 isa graphic of a three-dimensional model that represents a rectangularprism onto which local photos 400 and a remote photo 500 have beentexture mapped. A camera 720 may be manipulated by the user to controlhow three-dimensional model 710 is viewed. User 182 may use touch screen180 to navigate the three-dimensional model 710.

In some embodiments, camera model information used to construct thethree-dimensional model may include information about a viewing angle ofthe camera when the camera took the photographic image and a focallength of the camera. Such information may be of help in allowing serverphotogrammetry module 160 or mobile photogrammetry module 144 tointegrate the photo into the 3D model at 3D model application 136 byaiding in making initial guesses about how to position the images fortexture mapping in the context of photogrammetry.

Referring back to FIG. 1, touch screen 180 may offer a variety of inputoptions to user 182. For example, it may offer multitouch input. Suchmultitouch input may include gestures such as pinching and swiping, toperform features such as zooming and rotating. In general, touch screen180 may allow users to touch discrete points, or may allow for gestureswith a moving point of contact in order to obtain input to navigate thethree-dimensional model 710.

In general, the three-dimensional model provides two-dimensional viewsof the three-dimensional model, if touch screen 180 is a standardtwo-dimensional touch screen 180. However, as discussed below inconnection with FIGS. 10A and 10B, some embodiments provide for stereophotography and 3D screens that provide a three-dimensional effects inconjunction with the 3D model.

Various input from the touch screen allows the user to navigate 3D model710. By performing various single-touch and multi-touch gestures, touchscreen 180 allows various navigation actions including rotation,translation, revolution, and zooming. Additionally, various gesturesallow control over the model including establishing a position of thethree-dimensional model, a scale of a plurality of axes of thethree-dimensional model, and an orientation of the three-dimensionalmodel. In general, these changes can be made by touch or multi-touchgestures, including combinations of single-point touches, swipes,pinches, and multi-touch gestures incorporating an anchored finger and aswiping finger.

FIG. 8 is a screenshot that shows how various two dimensional images maybe dragged onto the side of a three-dimensional solid to texture mapthem onto the sides of the solid. In FIG. 8, touch screen 180 displayslocation data 810, such as “First and Elm”, that is associated with themodel. In this case, “First and Elm” is descriptive of a streetintersection that provides for the location of a building model thathelps select relevant images to map onto the building model 820, whichin this case is a rectangular prism. In this case, remote photo 500 haveselected because it is also associated as being located at “First andElm”. Local photo 400 represents a façade of one side of the rectangularsolid of building model 820, while remote photo 500 represents anoverhead view. By using touch screen 180, user 182 may tap local photo400 and then tap a front side of building model 820 to associate thelocal photo with that side, or alternatively may swipe a finger from thelocation of local photo 400 to the front side of building model 820 toassociate local photo 400 with the front side. Similar touch screen 180input may be used to associate the remote photo 500 with the top side ofbuilding model 820. Alternatively, the camera models associated withlocal photo 400 and remote photo 500 may be used to correlate torespective facades.

FIG. 9 is a screenshot that shows how the corners of a building in atwo-dimensional image may be mapped onto the corners of athree-dimensional solid to provide constraints. Such mapping providesfor more fine-tuned control over associating local image 400 withbuilding model 820. By altering how the building model is constrained tolocations in the photos, photogrammetry may be used to update thegeometry of the three dimensional model (and possibly also the cameramodels associated with the photos.)

FIG. 10A illustrates how stereo cameras may be used to obtain 3Dphotography, and FIG. 10B illustrates how a 3D model may be depicted ona 3D screen. In FIG. 10A, camera 130A obtains a left stereo view 1010and camera 130B obtains a right stereo view 1020. Together, the stereophotography may be combined to yield a three-dimensional perspective forbuilding views. Such a three-dimensional perspective may be portrayed ona three-dimensional screen as a 3D screen view 1050. Such a 3D screenview may require the use of special glasses to support a 3D view, or mayuse appropriate screen technologies to lend the screen a 3D aspectwithout requiring glasses. 3D screens use special optical effects toprovide the illusion of depth, and can allow users to perceive the 3Dmodel as being genuinely three-dimensional, rather than a projectiononto a two-dimensional screen, as in other embodiments.

FIG. 11 illustrates an interface for a building modeler, in anembodiment. The building modeler allows the user to alter the model bydefining solids. In the building modeler, building templates 1120buttons provide example shapes that may be dragged onto 3D map 1130 byusing appropriate touch screen 180 gestures. The building templates 1120allow the user to place building solids 1110 by tapping locations on 3Dmap or by swiping from the building template 1120 button to the desiredlocation. Then, the user may use appropriate gestures to performmanipulations on the building solids, such as rotation, translations,resizing, reflection and so on. Once the user has constructed a 3D map1130 by using building modeler, the 3D model may have images mapped ontoit as in FIG. 7.

Thus, embodiments incorporate a variety of features in new andadvantageous ways. Embodiments provide for easily obtaining images usingcameras coupled to a mobile device. As the images are obtained, cameramodel information is associated with them. The camera model informationmay include information including the orientation and location of thecameras. Embodiments combine local images with additional images thathave fields of view that overlap with the local images and also havecamera model information. To provide more information to allowphotogrammetry functionality (that may be provided at the mobile device,in a front-end implementation, or at a server, in a back-endimplementation), the user can show where points on the images map topoints on three-dimensional models.

By combining the information from the camera models and the user-setpoints, photogrammetry algorithms at the front-end or the back-endtexture map the images onto solids which represent buildings on a map.Similarly, images may be mapped onto walls indoors, in some embodiments.Given a sufficient set of images, the front-end or the back-end canobtain coverage for all surfaces in a field of view, providing a userwith access to a 3D simulation of a virtual environment based on thephotogrammetry. Additionally, stereophotography and a 3D screen in someembodiments may offer a true 3D representation of the model.

Thus, embodiments represent an effective way of obtaining and organizingimage data which is representative of an area, and combining the data toprovide and alter a 3D simulation of the area.

The Summary and Abstract sections may set forth one or more but not allexemplary embodiments of the present invention as contemplated by theinventor(s), and thus, are not intended to limit the present inventionand the appended claims in any way.

The present invention has been described above with the aid offunctional building blocks illustrating the implementation of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have been arbitrarily defined herein for the convenienceof the description. Alternate boundaries can be defined so long as thespecified functions and relationships thereof are appropriatelyperformed.

The foregoing description of the specific embodiments will so fullyreveal the general nature of the invention that others can, by applyingknowledge within the skill of the art, readily modify and/or adapt forvarious applications such specific embodiments, without undueexperimentation, without departing from the general concept of thepresent invention. Therefore, such adaptations and modifications areintended to be within the meaning and range of equivalents of thedisclosed embodiments, based on the teaching and guidance presentedherein. It is to be understood that the phraseology or terminologyherein is for the purpose of description and not of limitation, suchthat the terminology or phraseology of the present specification is tobe interpreted by the skilled artisan in light of the teachings andguidance.

The breadth and scope of the present invention should not be limited byany of the above-described exemplary embodiments, but should be definedonly in accordance with the following claims and their equivalents.

What is claimed is:
 1. A computer-implemented method for alteringthree-dimensional models based on two-dimensional images, comprising:(a) capturing, with a first camera of a mobile device, a localphotographic image with a first field of view; (b) detecting, usinginformation from a location sensor associated with the mobile device, afirst location and a first orientation of the first camera when thelocal photograph was captured in (a), wherein a first camera modelincludes the first location and the first orientation; (c) sending arequest to a server, separate from the mobile device, for an additionalphotographic image captured from a second camera remote from the mobiledevice, the additional photographic image having a second field of viewoverlapping with the first field of view; (d) receiving, from theserver, the additional photographic image and a second camera modelincluding a location and an orientation of the second camera when theadditional photographic image was captured; (e) receiving a userconstraint input by a user of the mobile device to be applied within athree-dimensional model, the user constraint associating an imageposition on the local photographic image with a corresponding positionwithin the three-dimensional model; and (f) applying a photogrammetryalgorithm to alter the three-dimensional model based, at least in parton, the first and second camera models and the user constraint such thatthe local photographic image is integrated into the three-dimensionalmodel in a manner that locates the image position at the correspondingposition within the three-dimensional model.
 2. The method of claim 1,further comprising the steps of: (g) determining whether the user wantsto add another constraint; and (h) when the user wants to add moreconstraints as determined in (g), repeating the receiving and theapplying with respect to the additional constraint.
 3. The method ofclaim 1, further comprising using a touch screen coupled to the mobiledevice to accept touch screen input from the user that allows the userto navigate through the three-dimensional model after it has beenaltered, wherein the touch screen allows multitouch input to navigatethe three-dimensional model.
 4. The method of claim 3, wherein the touchscreen allows the user to use gestures involving a moving point ofcontact to navigate the three-dimensional model.
 5. The method of claim3, wherein the touch screen input controls a navigation camera view ofthe three-dimensional model that provides a two-dimensional view of thethree-dimensional model, wherein the touch screen allows one or more ofrotation, translation, revolution, and zooming by the navigation cameraview with respect to the three-dimensional model.
 6. The method of claim1, wherein the camera model for each camera further comprises a viewingangle of the camera when the camera took the photographic image and afocal length of the camera, the viewing angle and focal lengthinformation being used in the altering of the three-dimensional model.7. The method of claim 1, further comprising displaying thethree-dimensional model on a three-dimensional display coupled to themobile device.
 8. The method of claim 1, wherein the user constraint isapplied to a set of images that constitute stereo photography such thatat least one image position for the set of images is associated with atleast one corresponding position within the three-dimensional model. 9.The method of claim 1, wherein at least steps (c)-(f) are carried out byan application on the mobile device, the application separate from a webbrowser on the mobile device.
 10. A system for alteringthree-dimensional models based on two-dimensional images the systemcomprising: a mobile device including a first camera configured tocapture a local photographic image with a first field of view and alocation sensor configured to capture a first location and a firstorientation of the first camera when the local photographic image iscaptured, wherein a first camera model includes the first location andthe first orientation, the mobile device further including one or moreprocessors and associated memory, the memory storing instructions that,when implemented by the one or more processors, configure the mobiledevice to implement: a server image request module, configured to send arequest to a server, separate from the mobile device, for an additionalphotographic image captured from a second camera remote from the mobiledevice, the additional photographic image having a second field of viewoverlapping with the first field of view; a server image receipt moduleconfigured to receive, from the server, the additional photographicimage and a second camera model including a location and an orientationof the second camera when the additional photographic image wascaptured; a user constraint input module configured to receive a userconstraint input by a user of the mobile device to be applied within athree-dimensional model, the user constraint associating an imageposition on the local photographic image with a corresponding positionwithin the three-dimensional model; and a photogrammetry moduleconfigured to apply a photogrammetry algorithm to alter thethree-dimensional model based, at least in part on, the first and secondcamera models and the user constraint such that the local photographicimage is integrated into the three-dimensional model in a manner thatlocates the image position at the corresponding position within thethree-dimensional model.
 11. The system of claim 10, wherein the userconstraint input module and photogrammetry module are further configuredto determine whether the user wants to add more user constraints. 12.The system of claim 10, further comprising a touch screen coupled to themobile device configured to accept touch screen input from the user thatallows the user to navigate through the three-dimensional model after ithas been altered, wherein the touch screen allows multitouch input tonavigate the three-dimensional model.
 13. The system of claim 12,wherein the touch screen allows the user to use gestures involving amoving point of contact to navigate the three-dimensional model.
 14. Thesystem of claim 12, wherein the touch screen input controls a navigationcamera view of the three-dimensional model that provides atwo-dimensional view of the three-dimensional model, wherein the touchscreen allows one or more of rotation, translation, revolution, orzooming by the navigation camera view with respect to thethree-dimensional model.
 15. The system of claim 10, wherein the cameramodel for each camera further comprises a viewing angle of the camerawhen the camera took the photographic image and a focal length of thecamera, the viewing angle and focal length information being used in thealtering of the three-dimensional model.
 16. The system of claim 10,further comprising a three-dimensional display coupled to the mobiledevice configured to display the three-dimensional model.
 17. The systemof claim 10, further comprising: at least one application, on the mobiledevice, that includes the server image request module, the server imagereceipt module, the user constraint input module, and the photogrammetrymodule; and a web browser on the mobile device that is separate from theat least one application.
 18. A non-transitory computer readable storagemedium having instructions stored thereon that, when executed by one ormore processors, cause the one or more processors to execute a methodfor altering three-dimensional models based on two-dimensional images,the method comprising: (a) capturing, with a first camera of a mobiledevice, a local photographic image with a first field of view; (b)detecting, using information from a location sensor associated with themobile device, a first location and a first orientation of the firstcamera when the local photograph was captured in (a), wherein a firstcamera model includes the first location and the first orientation; (c)sending a request to a server, separate from the mobile device, for anadditional photographic image captured from a second camera remote fromthe mobile device, the additional photographic image having a secondfield of view overlapping with the first field of view; (d) receiving,from the server, the additional photographic image and a second cameramodel including a location and an orientation of the second camera whenthe additional photographic image was captured; (e) receiving a userconstraint input by a user of the mobile device to be applied within athree-dimensional model, the user constraint associating an imageposition on the local photographic image with a corresponding positionwithin the three-dimensional model; and (f) applying a photogrammetryalgorithm to alter the three-dimensional model based, at least in parton, the first and second camera models and the user constraint such thatthe local photographic image is integrated into the three-dimensionalmodel in a manner that locates the image position at the correspondingposition within the three-dimensional model.
 19. The non-transitorycomputer readable storage medium of claim 18, wherein the camera modelfor each camera further comprises a viewing angle of the camera when thecamera took the photographic image and a focal length of the camera, theviewing angle and focal length information being used in the altering ofthe three-dimensional model.
 20. The non-transitory computer readablestorage medium of claim 18, wherein at least steps (c)-(t) are carriedout by an application on the mobile device, the application separatefrom a web browser on the mobile device.